Agnostically Learning Halfspaces with Margin Errors

We describe and analyze a new algorithm for agnostically learning half-spaces with respect to the margin error rate. Roughly speaking, this corre-sponds to the worst-case error rate after each point is perturbed by a noisevector of length at most μ. Margin based analysis is widely used in learningtheory and is considered the most successful theoretical explanation for thestatistical properties of several learning algorithms, such as Support VectorMachines and AdaBoost. The proposed algorithm can learn n-dimensionalhalfspaces in time poly(n exp( 1μ log( 1μ ))), for any distribution, where μ isthe margin parameter and the error rate of the learned classifier is at mostplus the margin error rate of the optimal halfspace. This improves overthe bound poly(n exp(( 1μ) 2 log( 1μ ))), derived by Ben-David and Simon [7].If the distribution over instances is uniform on the unit ball, we recoverthe poly(n1/ 4) complexity bound of Kalai et al [17]. Furthermore, the de-pendence on the dimension n in our complexity bound can be replaced bythe time required to calculate inner-products. This enables us to efficientlylearn halfspaces in possibly infinite dimensional Hilbert spaces by using theso-called kernel trick.